Nonconvexity of the generalized numerical range associated with the principal character

Suppose m and n are integers such that 1 ≤ m ≤ n. For a subgroup H of the symmetric group Sm of degree m, consider the generalized matrix function on m×m matrices B = (bij) defined by d(B) = ∑ σ∈H ∏m j=1 bjσ(j) and the generalized numerical range of an n×n complex matrix A associated with d defined by W(A) = {dH(X∗AX) : X is n×m such that X∗X = Im}. It is known that W(A) is convex if m = 1 or if m = n = 2. We show that there exist normal matrices A for which W(A) is not convex if 3 ≤ m ≤ n. Moreover, for m = 2 < n, we prove that a normal matrix A with eigenvalues lying on a straight line has convexW(A) if and only if νA is Hermitian for some nonzero ν ∈ C. These results extend those of Hu, Hurley and Tam, who studied the special case when 2 ≤ m ≤ 3 ≤ n and H = Sm.